…dedup

Three findings from Cursor Bugbot + Seer on PR #791:

**Seer (MEDIUM): followSymlinks: true silently drops symlinks**

`readdir({withFileTypes: true})` uses lstat semantics — on a symlink,
`isSymbolicLink()` returns true while `isFile()` and `isDirectory()`
both return false. The existing `processEntry` only routed entries
via `isDirectory()` / `isFile()`, so a symlink with `followSymlinks:
true` passed the skip check but then fell through and returned null.
The `inodeKey`/`visitedInodes` cycle detection in `maybeDescend` was
unreachable dead code.

Fix: when a symlink is encountered with `followSymlinks: true`,
`stat()` (not `lstat`) the target to learn its real type, route to
`maybeDescend` or `tryYieldFile` accordingly. Broken symlinks and
stat errors are silently skipped (matches ripgrep's behavior).
Circular symlinks are broken by the existing inode-based cycle
detection.

Tests: 4 new symlink cases in `walker.test.ts` covering default-skip,
follow-through, circular cycle detection, and broken symlinks.

**Cursor #2 (MEDIUM): concurrent grep workers share stateful RegExp**

When a pre-compiled `/gm` RegExp was passed to `grepFiles`,
`ensureGlobalMultilineFlags` returned it unchanged. Concurrent
`readAndGrep` workers then shared the regex's `lastIndex` property.
Strictly speaking this was safe today — JS is single-threaded and
the match loop has no `await`, so workers can't actually interleave
mid-iteration — but any future `await` inside the loop would turn it
into a silent match-dropping bug.

Fix: always clone the regex at the start of `readAndGrep` via `new
RegExp(src, flags)`. ~1µs per file, eliminates the foot-gun. Added a
regression test that exercises the shared-regex path with multiple
concurrent workers.

**Cursor #1 (LOW): shared utilities duplicated between grep and glob**

`compileMatchers`, `matchesAny`, `basenameOf`, `walkerRoot`, and
`joinPosix` were duplicated between `src/lib/scan/grep.ts` and
`glob.ts`. Extracted to new internal module `src/lib/scan/path-utils.ts`
(not part of the public barrel — implementation detail). Net LOC
reduction: ~80 lines across the two files.

## Test plan

- [x] `bun run typecheck` — clean
- [x] `bun run lint` — clean (1 pre-existing warning)
- [x] `bun test test/lib test/commands test/types` — **5443 pass, 0 fail** (+5 from this commit)
- [x] `bun test test/isolated` — 138 pass
- [x] `bun run bench --size large` — no regression on `scanCodeForDsns` (312ms) or `scan.grepFiles` (278ms)

## Summary

Two regex-level optimizations to narrow the perf gap with ripgrep on our
pure-TS `collectGrep`/`grepFiles`. Follow-up to PR #791 and #797.

- **Literal prefilter** — ripgrep-style: extract a literal substring
from the regex source (e.g., `import` from `import.*from`), scan the
buffer with `indexOf` to locate candidate lines, only invoke the regex
engine on lines that contain the literal. V8's `indexOf` is roughly
SIMD-speed; skipping the regex engine on non-candidate lines is where
most of the win comes from.
- **Lazy line counting** — swapped `charCodeAt`-walk for `indexOf("\n",
cursor)` hops. 2-5× faster on the line-counting sub-loop because V8
implements `indexOf` in C++ without per-iteration JS interop.

## Perf impact (synthetic/large, 10k files, Bun 1.3.11, 4-core)

| Op | Before | After | Δ |
|---|---:|---:|---:|
| `scan.grepFiles` (DSN pattern) | 370 ms | **318 ms** | **−14%** |
| `detectAllDsns.cold` | 363 ms | **313 ms** | **−14%** |
| `detectDsn.cold` | 7.73 ms | **5.61 ms** | **−27%** |
| `scanCodeForFirstDsn` | 2.91 ms | **2.13 ms** | **−27%** |
| `scanCodeForDsns` | 342 ms | 333 ms | −3% (noise-equivalent) |
| `import.*from` uncapped (bench) | 1489 ms | **1178 ms** | **−21%** |

The DSN workloads improve because `DSN_PATTERN` extracts `http` as its
literal — most source files don't contain `http` at all, so the
prefilter short-circuits before the regex runs.

No regressions on any benchmark. Pure-literal patterns (e.g.,
`SENTRY_DSN`, `NONEXISTENT_TOKEN_XYZ`) continue through the whole-buffer
path unchanged.

## What changed

### New file: `src/lib/scan/literal-extract.ts` (~300 LOC)

Conservative literal extractor. Walks a regex source looking for the
longest contiguous run of literal bytes that every match must contain.
Bails out safely on top-level alternation, character classes, groups,
lookarounds, quantifiers, and escape classes.

Handles escaped metacharacters intelligently: `Sentry\.init` yields
`Sentry.init` (extracted via literal `\.` → `.`), while `\bfoo\b` yields
`foo` (escape `\b` is an anchor, not a literal `b`).

Exports:
- `extractInnerLiteral(source, flags)` — returns the literal, or null if
no safe extraction possible. Honors `/i` by lowercasing.
- `isPureLiteral(source, flags)` — true when the pattern IS a bare
literal with no metacharacters. Used by the grep pipeline to route
pure-literals to the whole-buffer path (V8's regex engine is
hyper-optimized for pure-literal patterns; the prefilter adds overhead
without benefit there).

### Modified: `src/lib/scan/grep.ts` (~240 LOC changes)

Three-way dispatch in `readAndGrep` based on the extracted literal:

1. **`grepByLiteralPrefilter`** (new) — regex with extractable literal +
`multiline: true`. Uses `indexOf(literal)` to find candidate lines, runs
the regex engine only on those. This is the main perf win.
2. **`grepByWholeBuffer`** — existing path, used for:
   - Pure-literal patterns (V8 handles them optimally)
- Patterns with no extractable literal (complex regex, top-level
alternation)
   - `multiline: false` mode (the fast path requires per-line semantics)

Also: replaced the `charCodeAt`-walk that counted newlines char-by-char
with an `indexOf("\n", cursor)` hop loop. Extracted `buildMatch(ctx,
bounds)` as a shared helper to bundle the match-construction arguments.

### Tests added

- `test/lib/scan/literal-extract.test.ts` — **39 tests** covering the
extractor's rules (escape handling, quantifier drop, alternation bail,
case-insensitive, minimum length).
- `test/lib/scan/grep.test.ts` — **7 new tests** for the prefilter fast
path: correctness vs whole-buffer, escaped-literal extraction,
case-insensitive flag, zero-literal-hit short-circuit, routing of pure
literals to whole-buffer, and alternation routing.

## Why this approach

From the ripgrep research (attached to PR #791): rg's central perf trick
is extracting a literal from each regex and prefiltering with SIMD
memchr. V8 doesn't expose SIMD directly but its
`String.prototype.indexOf` is compiled to a tight byte-level loop with
internal SIMD on x64 — functionally equivalent for our use case.

Three of the five techniques in the Loggly regex-perf guide were
evaluated:
- **Character classes over `.*`** — `DSN_PATTERN` already uses
`[a-z0-9]+`, no change needed.
- **Alternation order** — `DSN_PATTERN`'s `(?:\.[a-z]+|:[0-9]+)` is
already correctly ordered (`.` more common than `:` in DSN hosts);
swapping regressed perf by noise.
- **Anchors/word boundaries** — adding `\b` to `DSN_PATTERN` *regressed*
perf 2.8× on our workload. V8's existing fast character-mismatch
rejection on the first byte outperforms the boundary check overhead.

The remaining gap with rg is now primarily orchestration overhead
(async/await, `mapFilesConcurrent`, walker correctness features) rather
than regex speed. A worker-pool exploration may follow.

## Test plan

- [x] `bunx tsc --noEmit` — clean
- [x] `bun run lint` — clean (1 pre-existing warning in
`src/lib/formatters/markdown.ts` unrelated to this PR)
- [x] `bun test --timeout 15000 test/lib test/commands test/types` —
**5610 pass, 0 fail** (+58 new)
- [x] `bun test test/isolated` — 138 pass, 0 fail
- [x] `bun run bench --size large --runs 5` — all scan ops at or below
previous baseline
- [x] Manually verified semantic parity: `collectGrep` returns identical
`GrepMatch[]` on prefilter vs whole-buffer paths for patterns where the
prefilter fires

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…#805)

## Summary

Walker was 411ms p50 for 10k files on the synthetic/large fixture —
roughly 10× slower than a naive sync walk (50ms). Follow-up to PR #791
and #804.

Five targeted optimizations in the hot loop, all micro-benchmarked
before landing:

| Change | Measured win | End-to-end win |
|---|---|---|
| `readdirSync` instead of `readdir` | 105→45ms (−60ms) | dominant |
| `statSync` instead of `Bun.file().size` | −10ms per 10k files | |
| String concat for paths | path.join 7ms → concat 0.7ms (10×) | |
| `abs.slice(cwdPrefixLen)` for relPath | path.relative 9ms → slice
0.8ms (11×) | |
| Manual `lastIndexOf`-based extname | 9ms → 7ms (25%) | |

### Perf results (synthetic/large, 10k files, p50)

| Op | Before | After | Δ |
|---|---:|---:|---:|
| `scan.walk` | 411ms | **231ms** | **−44%** |
| `scan.walk.noExt` | 572ms | 448ms | **−22%** |
| `scan.walk.dsnParity` | 228ms | **138ms** | **−39%** |
| `scanCodeForDsns` | 323ms | 304ms | −6% |
| `detectAllDsns.cold` | 327ms | 308ms | −6% |
| `detectAllDsns.warm` | 27.9ms | 27.0ms | — |
| `scan.grepFiles` | 322ms | 316ms | noise |
| `scanCodeForFirstDsn` | 2.23ms | 2.05ms | — |

Walker ops are 22-44% faster. Downstream ops (grep, DSN scanner) benefit
less because their time is dominated by content scanning, not walking —
but still show consistent ~6% improvements.

## Why `readdirSync` is safe here

The sync vs async tradeoff usually favors async because blocking the
event loop is bad in general. But measured per-call cost matters:

| Metric | `readdirSync` on 3635 dirs |
|---|---|
| p50 | 11µs |
| p95 | 24µs |
| p99 | 35µs |
| max | 65µs |

65µs max block is trivial — `setTimeout(0)` latency in Node is ~4ms.
Blocking for 65µs never causes noticeable event-loop pauses. And we pay
~60µs of microtask overhead for each async readdir, which wipes out any
theoretical fairness benefit. Net: 2-3× faster per-dir on walks with
many small directories, which is every realistic CLI workload.

If this ever matters for a weird embedded use case, the optimization is
trivially reversible — the walker's public API is unchanged.

## mtime parity fix

Initial implementation regressed `detectAllDsns.warm` from 28ms → 304ms
because `statSync().mtimeMs` is a float (e.g. `1776790602458.1033`)
while `Bun.file().lastModified` is already an integer. The DSN cache
validator compares floored `sourceMtimes`, so un-floored floats caused
cache misses on every warm call. Fixed by flooring explicitly in
`tryYieldFile` — matches the same treatment already applied to
`onDirectoryVisit`'s dirMtimes.

## Walker v2 design notes

- **Sync readdir + async yield.** Readdir blocks briefly (~65µs max),
but the surrounding async-generator interface is preserved. Library
consumers that insert async work between iterations still get
cooperative scheduling.
- **Path ops inlined.** `path.join` / `path.relative` / `path.extname` +
`toLowerCase` are all replaced with manual string ops in the hot loop.
On Windows, `normalizePath` is still applied via `replaceAll(NATIVE_SEP,
"/")` — the POSIX fast path uses a cached `POSIX_NATIVE` module
constant.
- **`WalkContext.cwdPrefixLen`** precomputed once per walk, used to
slice relative paths from absolute paths.
- **Cached module constants** `NATIVE_SEP` and `POSIX_NATIVE` avoid
per-call `path.sep` property lookups in the hot loop.

## What this PR does NOT change

- `WalkEntry` shape: preserved identically (absolutePath, relativePath,
size, mtime, isBinary, depth).
- `WalkOptions` contract: no new options, no semantics changes.
- Symlink handling, `followSymlinks`, cycle detection via
`visitedInodes`: unchanged.
- IgnoreStack + gitignore semantics: unchanged.
- `onDirectoryVisit`, `recordMtimes`, `abortSignal`: unchanged.

## Test plan

- [x] `bunx tsc --noEmit` — clean
- [x] `bun run lint` — clean (1 pre-existing warning in
`src/lib/formatters/markdown.ts` unrelated)
- [x] `bun test --timeout 15000 test/lib test/commands test/types` —
**5640 pass, 0 fail**
- [x] `bun test test/isolated` — 138 pass
- [x] `bun test test/lib/scan/walker.test.ts` — 34 pass (incl. property
tests covering hidden files, symlinks, maxDepth, gitignore interaction)
- [x] `bun test test/lib/dsn/code-scanner.test.ts` — 52 pass (incl.
dirMtimes / sourceMtimes cache validation)
- [x] DSN count correctness verified end-to-end: 4 DSNs found on fixture
(matches pre-change count)
- [x] `bun run bench --size large --runs 5 --warmup 2` — results in
table above

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>